Methods and systems for determining relevance of documents

ABSTRACT

A method for screening documents. The order of the text in the documents can be changed in order to create additional documents. A vocabulary of words from the text in the documents can be built. The words can be coded as vectors. Vector relationships can be defined. The documents can be classified using the vector relationships.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation application to U.S. application Ser.No. 16/891,396 filed Jun. 3, 2020, which is a continuation applicationto U.S. application Ser. No. 16/203,102 filed Nov. 28, 2018, whichclaims the benefit of U.S. Provisional Application No. 62/621,404 filedJan. 24, 2018, which are incorporated by reference in their entireties.

BRIEF DESCRIPTION OF THE DRAWINGS

FIGS. 1A and 1B illustrate example processes for screening resumes,according to embodiments of the invention.

FIGS. 1C and 1D illustrate example systems for screening resumes,according to an embodiment of the invention.

FIG. 2 provides an example of word-to-vector encoding, according to anembodiment of the invention.

FIG. 3 provides examples of how vector algebra can be used with wordvectors, according to an embodiment of the invention.

FIG. 4 illustrates how the system can be flexible and trained to learndifferent goals and vocabularies, according to an embodiment of theinvention.

FIG. 5 illustrates an example of PDF to text transformation, accordingto an embodiment of the invention.

FIGS. 6A, 6B and 6C illustrates an example of tokenization, according toan embodiment of the invention.

FIGS. 7A, 7B and 7C illustrate an example of resume input and process,according to an embodiment of the invention.

FIG. 8 illustrates an example of data augmentation, according to anembodiment of the invention.

FIGS. 9A, 9B and 9C illustrate an example use of a deep neural network,according to an embodiment of the invention.

FIG. 10 illustrates an example of predicting candidate goodness,according to an embodiment of the invention.

DESCRIPTION OF EMBODIMENTS OF THE INVENTION

The following description illustrates example embodiments for screeningdocuments. Although resumes are used throughout this application asexample embodiments, those of ordinary skill in the art will see thatmany other types of documents may also be utilized. FIG. 1A illustratesa process for screening resumes, according to an embodiment. In 105,resumes can be entered. In some embodiments, the resumes can be enteredin pdf format and converted to text. In 107, augmentation can be done onthe resumes to create additional resumes. (These can be used, forexample, in training the system.) In 110, a vocabulary list can be builtfor the position of interest. In 112, each word in the vocabulary listcan be coded as a vector using word embedding. In 115, vectorrelationships can be defined. In 120, a classification (e.g., based on arelevance probability score) for each resume can be output.

FIG. 1B illustrates an overview process for screening resumes, accordingto an embodiment. In FIG. 1B, resumes can be classified in a learningprocess based on content learned from relevant samples. In 195, wordsfrom resumes are input. In 196, each word of the resumes can betransformed into a series of numbers (e.g., vectors), which can becalled embeddings. In 197, the deep neural network can learn the vectorvalues using the relationships that adjacent words have in classifyinggood and bad resumes. In 198, each resume can be transformed into a setof previously learned word vectors which can be used to determine aclassification score which reflect the possibility that a candidate isworth pursuing. In FIG. 1B, resumes can be classified in a learningprocess based on content learned from relevant samples. In 195, wordsfrom resumes are input. In 196, each word of the resumes can betransformed into a series of numbers (e.g., vectors), which can becalled embeddings. In 197, the deep neural network can learn the vectorvalues using the relationships that adjacent words have in classifyinggood and bad resumes. In 198, each resume can be transformed into a setof previously learned word vectors which can be used to determine aclassification score which reflect the possibility that a candidate isworth pursuing.

FIG. 1C illustrates a block diagram of an example system architecture100 implementing the features and processes described herein. Thearchitecture 100 may be implemented on any electronic device that runssoftware applications derived from compiled instructions, includingwithout limitation personal computers, servers, smart phones, mediaplayers, electronic tablets, game consoles, email devices, etc. In someimplementations, the architecture 100 may include one or more processors102, one or more input devices 104, one or more display devices 106, oneor more network interfaces 108, and one or more computer-readablemediums 110. Each of these components may be coupled by bus 112.

Display device 106 may be any known display technology, including butnot limited to display devices using Liquid Crystal Display (LCD) orLight Emitting Diode (LED) technology. Processor(s) 102 may use anyknown processor technology, including but not limited to graphicsprocessors and multi-core processors. Input device 104 may be any knowninput device technology, including but not limited to a keyboard(including a virtual keyboard), mouse, track ball, and touch-sensitivepad or display. Bus 112 may be any known internal or external bustechnology, including but not limited to ISA, EISA, PCI, PCI Express,NuBus, USB, Serial ATA or FireWire. Computer-readable medium 110 may beany medium that participates in providing instructions to processor(s)102 for execution, including without limitation, non-volatile storagemedia (e.g., optical disks, magnetic disks, flash drives, etc.), orvolatile media (e.g., SDRAM, ROM, etc.).

Computer-readable medium 110 may include various instructions 114 forimplementing an operating system (e.g., Mac OS®, Windows®, Linux). Theoperating system may be multi-user, multiprocessing, multitasking,multithreading, real-time, and the like. The operating system mayperform basic tasks, including but not limited to: recognizing inputfrom input device 104; sending output to display device 106; keepingtrack of files and directories on computer-readable medium 110;controlling peripheral devices (e.g., disk drives, printers, etc.) whichcan be controlled directly or through an I/O controller; and managingtraffic on bus 112. Network communications instructions 116 mayestablish and maintain network connections (e.g., software forimplementing communication protocols, such as TCP/IP, HTTP, Ethernet,etc.).

FIG. 1D illustrates a system 199 for screening documents, according toan embodiment. This system comprises: a PDF to text module 195, anaugmentation module 188, a build vocabulary module 190; a coding wordsmodule 185, a deep neural network module 180, and a classify resumesmodule 175.

FIG. 2 provides an example of how embeddings can encode and summarizerelevant features. For example, the word HOUSE can be turned into avector using word-to-vector encoding. The vectors can comprise latent orunknown vectors that the system determines are relevant. Mathmaticallyspeaking, embeddings can be mappings. Size can be a part of designingthe problem solution. Each word (e.g., 172,000 in the Englishvocabulary) can be a dimension. The features can be about a word'ssyntax and semantic context captured by learning the embeddings.

FIG. 3 provides examples of how vector algebra can be used with the wordvectors. In FIG. 3 , the algebra of vectors can mimic the correctinterpretation of English words, using latent features vectors (i.e.embeddings). As a very simply example, if the system recognizes that a“good” resume will have a sum of 12, then any combination of vectorvalues that reaches 12 can indicate that that resume is a good resume.So if two vectors have a value of 1 and 5, then the sum is 6, and thiswould not be a “good” resume. However, if two word vectors have a sum of13, then this would be considered a “good” resume.

The screening process can be utilized to determine the probability thata resume is of interest. For example, if 1000 resumes are submitted fora particular position, the screening process can provide a probabilityof relevance for each resume and rank the order of the resumes accordingto the probability of relevance rankings. In this way, a hiring personcan determine almost immediately which resumes are the most important.This may be valuable, for example, when it is important to make an earlycontact for a prime candidate for a position, rather than waitingseveral weeks or months to get to that particular resume in a list orpile of resumes. This process can also, for example, cut downsignificantly on the amount of time it takes to review resumes. Forexample, when a probability is given for each resume, a hiring personcan start reviewing the resumes with the highest probabilities first inorder to maximize the chances that the most relevant or desirable hiresare reviewed and interviewed first. The screening process can also, forexample, cut down on the time it takes to interview a candidate. Forexample, if a resume is determined to be a very good candidate, thefirst pre-screening interview can focus on language skills andpersonality, instead of qualifications, which can result in asignificant reduction of the time spent in the first interview.

In some embodiments, resumes that are determined to have a lowprobability of relevance can be reviewed, and if it is determined thatthey should have a higher probability of relevance, these can be fedinto the system in order to better train the system. In a similar way,resumes that are determined to have a high probability of relevance canbe reviewed, and if it is determined that they should have a lowerprobability of relevance, these can be fed into the system in order tobetter train the system.

FIG. 4 illustrates how the system can be flexible and trained to learndifferent goals and vocabularies. For example, with respect to thevocabulary setting, sample documents (e.g., resumes) in PDF format canbe input into the system. The PDF documents can be imported into thesystem, and the words can be tokenized. A dictionary of vocabulary wordscan be created. The vocabulary words can be used to embed an index.

With respect to the model training, two folders of PDF documents can becreated, one with examples of good candidates or resumes, and one withexamples of bad candidates or resumes. The PDF documents can beconverted to text. Sentences in the resumes can be pre-processed. Thewords can be mapped to embeddings. The data can then be augmented. Amodel code and performance metrics can be output with accuracy, recall(e.g., true positives divided by the sum of true positives and falsenegatives) and precision (e.g., true positives divided by the sum of thetrue positives and the false positives) information.

With respect to candidate predictions, three set of data may be input:resumes (in PDF format) to be scored, the relevant vocabulary list, andthe model. Each PDF document can be run through the model and aprobability of being a good candidate can be assigned. The output can bea prioritized list of candidates to interview.

Example Embodiments

Vocabulary Building. In some embodiments, the following process can beused to build the vocabulary.

-   -   1. Import resume file names into a dataframe.    -   2. For each line of the dataframe.        -   a. Read the resume file name.        -   b. Import the PDF file.        -   c. Transform the PDF into text and append it as a column of            the dataframe.        -   d. Process the text and append the list of words as a column            of the dataframe.        -   e. Calculate the number of words in each resume and append            it as a column to the dataframe.    -   3. Create a set including all the words in the resumes.    -   4. Count how many times each word appears.    -   5. Sort words by count in descending order.    -   6. Create a dictionary of vocabulary words using number to word        matching based on previous sorting.

PDF to Text Transformation. In some embodiments, the following processcan be used to transform the pdf document to text.

-   -   1. Create a list of words (e.g., English, French, any language)        present in all the resumes we are going to examine. Each word        can be called a “token”.    -   2. The resumes used as input can be stored in PDF, which can be        the format candidates use to submit the resume in order to apply        to a job. Because the PDF format may not allow immediate token        identification and manipulation, we can transform the PDF        documents into text documents using the example function set        forth in FIG. 5 which takes a PDF document as input and returns        a text version of the document as output. The function in FIG. 5        can be re-used at any stage of the process to transform a PDF        into text. For example, this process can be used on a batch of        resumes being used to train the model. This function can also be        used when a new pdf resume is received and checked to see if it        is good (e.g., a desirable candidate). In both these cases pdf        files need to be transformed into text. Of course, if a resume        is received in a format other than PDF, it can be easily        converted into PDF using standard PDF technology and then the        PDF document can be transformed into text.

Tokenization.

-   -   1. Once the PDF documents have been transformed into text, the        tokens (e.g., words) can be extracted. FIG. 6A illustrates an        example function where a text file is input, and a set of tokens        are output. The text can be split into words, sentences, and/or        paragraphs using a set of regular expressions and character        based rules (e.g., using Regex). This function can be re-used at        any stage of the process to tokenize text. For example, this        function can be used on a batch of resumes being used to train        the model, but can also be used when a new pdf resume is        received and is being checked for the probability of the resume        being a good one. In both cases, the text can be tokenized.    -   2. Once we have the tokens, we can count how many times each        word appears in the corpus (e.g., the set of documents        considered) as shown in FIG. 6B.    -   3. When we have the count of how many times each word appears in        the corpus (e.g., the body of resumes), we can sort those words        in decreasing order (e.g., the words with the highest count        appear first) as shown in FIG. 6C and we can assign each word        their ranking number (e.g., index) in this special sorted list.        If you have two words appearing the exact same number of times,        in some embodiments, they can be given subsequent numbers in        alphabetical order.

Training. The training procedure can assume there is a set of resume PDFdocuments that have been stored in two different folders. Folder “yes”can include candidates that have been deemed to be successful in thepast after a pre-screening hiring round by a hiring team of person(s).Folder “no” can include candidates who did not pass the pre-screeninghiring round by the hiring team of person(s). In some embodiment, thefollowing process can be used, For example, using Python as the codinglanguage, with a Keras library and a TensorFlow backend.

-   -   1. Import resume file names from the “yes'” folder into a        dataframe.    -   2. Create a column “proceed”.    -   3. For each resume append the value 1 in the column “proceed”.    -   4. Import resume file names from the “no” folder into a        dataframe.    -   5. For each resume append the value 0 in the column “proceed”.    -   6. For each line of the dataframe:        -   a. Read the resume file name.        -   b. Import the PDF file.        -   c. Transform the PDF into text and append it as a column of            the dataframe.        -   d. Process the text and append the list of words as a column            of the dataframe.        -   e. Transform list of word into a list of indexes using the            pre-built vocabulary and append it to the dataframe.    -   7. Split dataset into train and validation randomly (e.g., using        known machine learning techniques).    -   8. Augment data by using a sentence and/or paragraph shuffling        procedure. This can be done only for the train dataset in some        embodiments.    -   9. Append the additional resumes created in the previous step to        the training dataset.    -   10. Pre-process data to feed Deep Neural networks.    -   11. Train Deep Neural networks.

Resume Import and Processing. The example function of FIG. 7A can beused to import resumes and mark them as “good” (=1) or “bad” (=0). Wecan then proceed to transform the resume into a list of indexes usingthe vocabulary. This can be done using an example procedure join_cvshown in FIG. 7B, which can use a supporting function named encode_cv.Once we have transformed the resume into a list of word indexes based onthe predetermined vocabulary, as shown in FIG. 7C, we can proceed with atypical machine learning step of splitting our data into: 1) a trainingdataset, which can be used to determine the model parameters; and 2) avalidation dataset, which can be used to test the generalization of themodel. Once we have determined the training set, we can proceed with thedata augmentation technique.

Data Augmentation. Because machines learn from examples, it is quitehelpful to have more examples (e.g., hundreds or thousands or more). Byusing the fact that sentences resumes are quite independent, we cancreate “new” example resumes by shuffling sentences belonging to theresume around. This technique can improve the prediction power of themodel. For example, in FIG. D1 , the text document for the resume can besplit into segments by finding the end of the paragraph and/or sentence(e.g., in the code in FIG. 8 , we refer to these segments as“sentences”), which can be designated in the text by a return, which canbe designated in the text as “\n”. We can then use the Pythoninstruction ‘random.shuffle( )’ to shuffle the sentences around. Theshuffled sentences can then be joined together to create new exampleresumes that can help teach the computer. For example, if we found 7sentences in a particular resume, we could order those S1 to S7, andthen shuffle them randomly so that we create several more exampleresumes. Thus, the original resume could be ordered S1, S2, S3, S4, S5,S6, and S7. After shuffling and joining together the shuffled sentences,we could have several other example resumes made of the same sentences.An example could be the resume including sentences in the followingorder: S2, S4, S7, S5, S1, S3, and S6. Once we have the new resumes, wecan turn them into tokens and encode them. This process can be helpfulwhen treating resumes as these documents usually include sentencesand/or paragraphs and/or other portions that are independent from eachother. The resulting ‘shuffled’ resumes are still resumes.

Deep Neural Network. We are now ready to do some pre-processing of theencoded resumes before feeding them to our Deep Neural Network.

A deep neural network (DNN) is an ANN (Artificial Neural Network) withmultiple hidden layers between the input and output layers.

https://en.wikipedia.org/wiki/Deep_learning#Deep_neural_networks. FIG.9A is an example of pre-processing that comprises: eliminatingnon-frequent words, padding the text to make all resume the same length,removing resumes that are too short, etc. At this point, we can createthe architecture of our deep neural network and start training it. Forexample, embeddings can be used so that word indexes can be mapped torandom vectors of a fixed size (e.g. 100, 200, 300). Those vectors canget their numbers fixed during the training procedure. FIG. 9Billustrates an example of a Deep Neural Network using convolutionallayers and word embeddings written in Keras. The embeddings value cancreate a matrix of a number of words in the resume×size of the embeddingvector.

The following code represents a Deep Neural Network written in Keras

Each row is a layer of the network gets input from the previous layer,applies the layer specific transformations and provides its output asthe following layer input.

 Specific trasformations are:  Embedding(vocab_size+1, 100,input_length=seq_len),  Transforms the resumes into a matrix with 100columns  Dropout(0.2),  Randomly deletes 20% of the output of theembedding layer   Convolution1D(12, 3, padding=‘same’,activation=‘relu’),   Applies a convolution operation to the output ofthe dropout layer and then applies a relu (rectified linear unit)function to it  Dropout(dof),  See above   MaxPooling1D( ),  Takes themaximum values of the output provided by the dropout layer  Convolution1D(12, 3, padding=‘same’, activation=‘relu’),  See above  Dropout(dof),  See above   MaxPooling1D( ),  See above   Flatten( ), Transforms the output of the maxpooling layer into a long flat vectorto be fed into the dense layer   Dense(10, activation=‘relu’), Transforms the output of flatten layer and condenses it into 10positive  numbers   Dropout(0.7),  See above   Dense(1,activation=‘sigmoid’)])

Transforms the output of dropout into 1 positive between 0 and 1 whichrepresents the probability of the resume to be a ‘good’ one. FIG. 9Cillustrates an example transformation of a resume sentence into a set ofembeddings using a word index vocabulary. The embeddings values can beinitialized randomly. The Deep Neural Network algorithm can find theones that best fit our proposed purpose during the training procedureusing an optimization algorithm such as, for example, the StochasticGradient Descent.

Stochastic gradient descent (often shortened to SGD), also known asincremental gradient descent, is a stochastic approximation of thegradient descent optimization and iterative method for minimizing anobjective function that is written as a sum of differentiable functions.

https://en.wikipedia.org/wiki/Stochastic_gradient_descent. When we arefinished training the model, we can save both the embeddings value andthe model multipliers into a file.

Predicting Candidate “Goodness”. To predict how a new candidate willperform during pre-screening steps, we can load the model file and passto it the PDF to get a prediction. The prediction process set forthbelow can be called scoring and for each resume it can return thepredicted probability of the candidate passing an HR department'spre-screening tests.

-   -   1. Import resume into a dataframe    -   2. For each line of the dataframe.        -   a. Read the resume file name.        -   b. Import the PDF file.        -   c. Transform the PDF into text and append it as a column of            the dataframe.        -   d. Process the text and append the list of words as a column            of the dataframe.        -   e. Transform list of word into a list of indexes using the            pre-built vocabulary and append it to the dataframe.        -   f. Apply same pre-processing steps as was done in the Deep            Neural Network section.    -   3. Load model.    -   4. Feed resume to model.    -   5. Get prediction.    -   6. Output prediction to .csv file

Once the PDF has been transformed into a sequence of indexes andpre-processed as described above, the scoring procedure comes frommultiple transformations of the word vectors using model parametersfound during training for the neural network functions. In verysimplified terms, this transformation can be represented as y(x)=f(x),where: y can be a predicted probability of a candidate being “good”; fcan be the set of parametrized transformations happening as result ofoptimized neural network parameters determined by the Deep NeuralNetwork algorithm during the model training procedure; and x can be theencoded resume (which can be translated into embeddings during thescoring procedure).

predictions=NN.predict(X_cand, verbose=1)

The output can then reshaped to a nice .cvs format as illustrated inFIG. 10 .

While various embodiments have been described above, it should beunderstood that they have been presented by way of example and notlimitation. It will be apparent to persons skilled in the relevantart(s) that various changes in form and detail can be made thereinwithout departing from the spirit and scope. In fact, after reading theabove description, it will be apparent to one skilled in the relevantart(s) how to implement alternative embodiments. For example, othersteps may be provided, or steps may be eliminated, from the describedflows, and other components may be added to, or removed from, thedescribed systems. Accordingly, other implementations are within thescope of the following claims.

In addition, it should be understood that any figures which highlightthe functionality and advantages are presented for example purposesonly. The disclosed methodology and system are each sufficientlyflexible and configurable such that they may be utilized in ways otherthan that shown.

Although the term “at least one” may often be used in the specification,claims and drawings, the terms “a”, “an”, “the”, “said”, etc. alsosignify “at least one” or “the at least one” in the specification,claims and drawings.

Finally, it is the applicant's intent that only claims that include theexpress language “means for” or “step for” be interpreted under 35U.S.C. 112(f). Claims that do not expressly include the phrase “meansfor” or “step for” are not to be interpreted under 35 U.S.C. 112(f).

1. A method for screening documents, comprising: changing the order ofthe text in the documents in order to create additional documents;building a vocabulary of words from the text in the documents; codingthe words as vectors; defining vector relationships; and classifying thedocuments utilizing the vector relationships.